Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Noisy Matrix Decomposition

We propose an efficient ADMM method with guarantees for high-dimensional problems. We provide explicit bounds for the sparse optimization problem and the noisy matrix decomposition problem. For sparse optimization, we establish that the modified ADMM method has an optimal regret bound of O(s log d/T ), where s is the sparsity level, d is the data dimension and T is the number of steps. This matches with the minimax lower bounds for sparse estimation. For matrix decomposition into sparse and low rank components, we provide the first guarantees for any online method, and prove a regret bound of Õ((s + r)β(p)/T ) + O(1/p) for a p × p matrix, where s is the sparsity level, r is the rank and Θ( √ p) ≤ β(p) ≤ Θ(p). Our guarantees match the minimax lower bound with respect to s, r and T . In addition, we match the minimax lower bound with respect to the matrix dimension p, i.e. β(p) = Θ( √ p), for many important statistical models including the independent noise model, the linear Bayesian network and the latent Gaussian graphical model under some conditions. Our ADMM method is based on epochbased annealing and consists of inexpensive steps which involve projections on to simple norm balls.